This application relates generally to the use of augmented reality to identify and display changes in objects in dynamic environments and, more particularly, to the use of augmented reality to provide information and direction to users operating in, manipulating or inspecting objects or structures in such dynamic environments and for identifying and displaying discrepancies in the position, configuration, or condition of such objects.
Augmented reality (AR) provides a view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, text, graphics, or video. In a typical AR application, a data processor reviews a camera-captured image for cues that trigger the display of additional information and images along with a display of the captured image.
While AR is useful for conveying information via a static display, it is particularly useful in dynamic situations and environments where images are captured and displayed in real-time and the information to be displayed using AR (hereinafter, “AR information”) changes. The ability to provide constant update to the AR information in response to changes in the environment and location and relative positioning of the user's display provides great utility in various applications including construction, repair, maintenance, and safety.
A particularly significant example of a dynamic spatial environment is the space on board a large ship. Not only does the ship itself move, its structure is flexible so that the position of a particular compartment or supporting structure in one part of the ship may change relative to other parts of the ship. Similar dynamic behavior can occur in tall buildings, construction sites, outdoor processing plants, roadways, bridges, etc.
In addition to changes in environmental structure, dynamic environments experience changes in an essentially infinite number of conditions. These can be as simple as changes in measurable atmospheric conditions (e.g., temperature and pressure) or as complex as changes in the surface topography of a metal sheet undergoing a welding operation.
Such changes can have significant impact on manufacturing and maintenance operations. Of particular interest for the present invention is the impact on inspection of equipment and structures for discrepancies and non-conformities.
A manufactured item or product can include assemblies, sub-assemblies, material, components, accessories and parts that may require inspection for discrepancies or non-conformities. A discrepancy is any variation between the item and its ordering data, drawings, specifications, procedures or other invoked requirements placed on that item, including the absence of required hardware or software (e.g., test data, certifications or approved drawings). Such discrepancies may also be discovered at any time during receipt, manufacturing, installation or post-installation of the item. Discovering a discrepancy generally involves, but is not limited to, a visual inspection of an item that can include: observing obvious physical damage; comparing an item's dimensions, construction, materials or other observable physical characteristic with the item's written specifications or drawings; checking the item's correct location and installation in a larger system. Often a certain level of expertise obtained through years of experience is required to determine whether an item or assembly of items has a discrepancy. However, for many industries that are undergoing a high rate of retirement of experienced workers, it is becoming increasingly difficult to maintain the level of experience that is required.
In addition to discovering a discrepancy in an item or assembly of items, the discrepancy must be recorded and reported in a timely manner to reduce the impact on production schedules or the proper operation of the item, either by itself or as an element of a complete system. Discovering, recording and reporting discrepancies currently often involves transporting hardcopy drawings or data and tools for measurement, for example, from one location to another, a process which can be inefficient and prone to error, damage or loss.
Embodiments of the present invention provide systems for inspecting, displaying and reporting discrepancies using AR. These systems can provide a worker charged with performing an inspection all of the information needed to inspect an item or items and report discrepancies, without the use of hardcopy plans, drawings, requirements or specifications. The invention can also be used to compensate for lack of experience by providing a visual, step-by-step description of a particular inspection procedure. In some embodiments, a worker identifying discrepancies in an item or system of items can electronically scan the item, via camera means, collect data from the scan, and compare that data to a visualized model representation of that item or system of items in real time. The worker may also be able to assign attributes to that model at the same time. Additionally, the results of the inspection can be recorded and reported, without paper forms, in some cases using the same device used to capture information on the item being inspected.